Variations on the Peer Review Theme 
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Prof. Guston has ably summarized the essentials concerning regulatory peer review. This 
note addresses a few small issues that have recently arisen. 

Regulatory peer review provides scientific advice, but science advice includes more 
than peer review. Guston noted this difference, but it has become more significant 
recently with the invention of what EPA calls “peer consultation .” 1 The Agency 
distinguishes “consultation” from “review" largely on the basis of status in the process of 
developing regulatory documents. In practice, the distinction may best be understood as 
the difference between reviewing early and final draft versions of documents. Peer review 
typically concerns regulatory documents — usually risk or policy analyses or components 
of such analyses — that are in a final draft stage. They frequently represent nearly settled 
thinking on the scientific issues involved. Consultation typically concerns such documents 
that are settled enough to permit review, but in which the analysis is still in flux. 

“Science advice” includes both of these, and can also refer to consideration of processes, 
methods, approaches, etc., that are not described in regulatory documents. We note that a 
favorable external peer review has increasingly come to represent a stamp of approval on 
agency positions, and thus something not necessarily desirable (to policy makers) to seek 
while positions are still evolving. Members of the Science Advisory Board have long 
noted that their work can be of greater use to the Agency if their opinions are sought early 
in the process of policy formation, rather than late, and asked to be so consulted. Peer 
consultation helps satisfy this desire while avoiding the risk of receiving an unfavorable 
review of a final Agency position. 

The dual purposes of regulatory' peer review tempt agencies to perform faux 
(cosmetic) peer reviews. As Guston notes, the ultimate purposes of regulatory peer 
review differ from those of reviews for publication and for granting agencies. Publication 
and grant peer reviews are intended to reject the unfit, to identify the minority of 

manuscripts or grant applications that will be published or funded. (To be accurate, these 
peer reviews usually result in each document being given a rank-ordered recommendation, 
e.g., “Definitely publish,” “Publish, with revisions.” “Consider publishing after revisions, 
if space is available.” “Not suitable for publication.”) Typically, the major research 
granting agencies fund fewer than one in five applications; the manuscript acceptance rate 
for journals varies, but seldom do half of those submitted see publication in the first 
journal to which they are submitted. 


See the accompanying memo from Don Barnes that describes the SAB usage of these terms. 
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Regulatory peer review and science advice share with these others a quality-control 
purpose, although this is almost never expressed as acceptance or rejection of a document 
in toto. The usual response consists of a few general comments on the document as a 
whole, followed by detailed comments (editorial as well as substantive) on individual 
passages or sections of the document. It is now almost always that case that regulatory 
science documents include at least a competent review of the relevant literature, and often 
an excellent one. So there is almost always something for reviewers to praise and accept. 
More likely to gamer critical commentary are key assumptions and the reasoning used to 
justify them, and inferences drawn from the scientific observations. Failure to identify key 
assumptions and / or inferences will today almost always engender strong criticism from a 
peer review, and may occasion rejection of the document. Yet sometimes these key 
assumptions are subtle, or require sophisticated scientific knowledge to understand. 

Having people who possess such knowledge part of the review process becomes very 
important to its utility. 

Peer reviews can go awry for a variety of reasons. One way is absence of particular 
scientific or technical expertise: those setting up the panel fail to include people with 
critical knowledge and understanding. Another way is a panel that is not representative in 
its societal (nonscientific) values; such a group can bias the review, one way or another. A 
third is inadequate framing of the questions asked or issues raised. The current 
controversies over regulatory peer review occur because of these possibilities. 

Policy makers learned very early on the political value of being able to claim peer-review 
approval of the analyses on which their decisions are based. Thus arose, as night follows 
day, the phenomenon of cosmetic peer review, /,<?., one set up only to approve an analysis 
— or to reject it, depending on the aims of those organizing the review. Each of the three 
sources of misadventure noted above can be manipulated by those so inclined, and we can 
assume that each has been so manipulated one time or another. 
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